[release][train] Adding py3.13 ray-ml image with torchft-nightly#63587
Open
elliot-barn wants to merge 11 commits into
Open
[release][train] Adding py3.13 ray-ml image with torchft-nightly#63587elliot-barn wants to merge 11 commits into
elliot-barn wants to merge 11 commits into
Conversation
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces support for Python 3.13 across the build and release infrastructure, including updates to Buildkite configurations, dependency lock files, and BYOD requirements. It also adds a new suite of nightly training ingest benchmarks for Python 3.13. Feedback was provided regarding a potential typo in a configuration flag, redundant dependency declarations in the new requirements file, and inconsistent argument formatting in the release test definitions.
I am having trouble creating individual review comments. Click here to see my feedback.
release/release_tests.yaml (1967)
anyscale_sdk_2026: true appears to be a typo. This flag is typically anyscale_sdk_v2: true in Ray release tests. Please verify if this is the intended key.
anyscale_sdk_v2: truerelease/ray_release/byod/requirements_ml_byod_3.13.in (43-44)
Both torchft==0.1.1 and torchft-nightly are listed. Since the pull request aims to include the nightly version, the stable version is redundant and may cause installation conflicts. It should be removed.
torchft-nightly
release/release_tests.yaml (2038)
The arguments --skip_train_step True and --skip_validation_at_epoch_end True use a space-separated format, which is inconsistent with the --arg=value format used in all other variations of this test (e.g., lines 1989, 2058). Using the consistent format improves maintainability and avoids potential parsing issues.
script: RAY_TRAIN_V2_ENABLED=1 python train_benchmark.py --task=image_classification --dataloader_type=ray_data --num_workers=16 --skip_train_step=True --skip_validation_at_epoch_end=True --image_classification_data_format=s3_url
|
This pull request has been automatically marked as stale because it has not had You can always ask for help on our discussion forum or Ray's public slack channel. If you'd like to keep this open, just leave any comment, and the stale label will be removed. |
github-actions
Bot
added
the
stale
Add a self-contained raydepsets depset (release_ml_torchft_tests.depsets.yaml) that compiles the Ray ML release-test dependencies with torchft-nightly layered on top, producing release/ray_release/byod/ml_torchft_py3.13.lock for py3.13 / cu128. It is installed onto the core Ray CUDA image via byod_ml_torchft.sh, so torchft release tests no longer depend on the published py3.13 ray-ml image (which fails to build due to dask/nixl py3.13 gaps). Decouple from the in-progress published py3.13 ray-ml image work by reverting the buildkite image/release steps, ray-images.json, the gpu BYOD py3.13 allowance, and the ml-base-extra-testdeps py3.13 depset + locks, and by removing torchft from the shared requirements_ml_byod_*.in files. torchft now lives only in the dedicated requirements_ml_torchft.in. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
…/github.com/ray-project/ray into elliot-barn-add-torchft-to-ml-release-image
…ft.txt Align the py3.13 torchft release-image depset with master after the torch 2.9.0 upgrade (#63361): - Bump requirements_ml_byod_3.13.in to torch==2.9.0 and drop the stale triton==3.3.0 pin (torch 2.9.0 pulls triton==3.5.0 transitively), matching the py3.13 constraint and ML requirement files. - Source torchft from the canonical python/requirements/ml/py313/torchft.txt (torchft-nightly==2026.5.15, torch-2.9.0-compatible) instead of a separate requirements_ml_torchft.in, so there is a single torchft pin. - Regenerate ml_torchft_py3.13.lock -> torch==2.9.0+cu128 / torchaudio 2.11.0+cu128 / triton 3.5.0; verified idempotent so raydepsets --check passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Add a minimal reference release test showing how to run a release test on the
torchft Ray ML image variant. It uses the core Ray CUDA image (py3.13) with the
torchft dependency lock installed on top:
cluster:
anyscale_sdk_2026: true
byod:
type: cu123
post_build_script: byod_ml_torchft.sh
python_depset: ml_torchft_py3.13.lock
The workload imports torch (2.9.0) + torchft and runs a short Ray Train v2 +
torchft linear training loop to prove the image works end to end. Validated
against the release schema (//release:test_config).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Setting byod.python_depset is sufficient: the BYOD image build automatically copies the lock in and runs `uv pip install --system --no-deps -r python_depset.lock` (release/ray_release/byod/build_context.py). The custom byod_ml_torchft.sh ran the identical command, so it installed the deps a second time for no reason. Remove byod_ml_torchft.sh and the post_build_script reference from the torchft_hello_world reference test; rely on python_depset alone. Validated with //release:test_config. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
github-actions
Bot
added
unstale
The py3.13 ML stack pinned sentencepiece==0.1.96 via python/requirements/ml/py313/train-test-requirements.txt, which flows into requirements_compiled_py3.13.txt. sentencepiece 0.1.96 ships no cp313 wheel, so `uv pip install --no-deps` falls back to building it from sdist, which needs cmake (absent in the BYOD base image) and fails. This only surfaced on the py3.13 torchft image; the py3.10/3.12 ML images have a cp310/cp312 wheel. Bump to sentencepiece==0.2.1 (first release with cp313 manylinux wheels), regenerate requirements_compiled_py3.13.txt (only the sentencepiece pin changes), and regenerate ml_torchft_py3.13.lock. No other py3.13 deplock pins sentencepiece, so the change is contained. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
creating a ray-ml py3.13 release test image with torchft-nightly
Creating a python 3.13 variation of training_ingest_benchmark-task=image_classification for full_training.jpeg and full_training.s3_url
release test run: https://buildkite.com/ray-project/release/builds/93976